Skip to content

[libressl] add download mirror#11949

Merged
strega-nil merged 15 commits intomicrosoft:masterfrom
Maximus5:libressl-mirror
Aug 7, 2020
Merged

[libressl] add download mirror#11949
strega-nil merged 15 commits intomicrosoft:masterfrom
Maximus5:libressl-mirror

Conversation

@Maximus5
Copy link
Copy Markdown
Contributor

Describe the pull request

Add alternative download mirror for libressl port.

When ftp.openbsd.org is not available, use ftp.fau.de.

Sample build output

1>-- [OVERLAY] Loading triplet configuration from: C:\SRC\vcpkg-ports\triplets\avira-x86-windows-static.cmake
1>-- Downloading https://ftp.openbsd.org/pub/OpenBSD/LibreSSL/libressl-2.9.1.tar.gz...
1>-- Downloading https://ftp.openbsd.org/pub/OpenBSD/LibreSSL/libressl-2.9.1.tar.gz... Failed. Status: 7;"Couldn't connect to server"
1>-- Downloading https://ftp.fau.de/openbsd/LibreSSL/libressl-2.9.1.tar.gz...
1>-- Extracting source C:/SRC/remediation-sdk/vcpkg/downloads/libressl-2.9.1.tar.gz

@JackBoosY JackBoosY added the category:port-feature The issue is with a library, which is requesting new capabilities that didn’t exist label Jun 16, 2020
Copy link
Copy Markdown
Contributor

@NancyLi1013 NancyLi1013 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please also bump the version as 2.9.1-3 in ports/libressl/CONTROL file?

Comment thread ports/libressl/portfile.cmake
Comment thread ports/libressl/portfile.cmake
@ghost
Copy link
Copy Markdown

ghost commented Jun 16, 2020

CLA assistant check
All CLA requirements met.

@Maximus5 Maximus5 requested a review from NancyLi1013 June 16, 2020 10:23
Comment thread ports/libressl/portfile.cmake Outdated
Comment thread ports/libressl/portfile.cmake Outdated
Comment thread ports/libressl/portfile.cmake
Update the deprecated functions or variables
@NancyLi1013
Copy link
Copy Markdown
Contributor

Hi @Maximus5
Have you tested feature tools?

It failed on x64-windows due to this:

-- Installing: F:/11949/vcpkg/packages/libressl_x64-windows/share/libressl/copyright
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x64-windows/debug/lib/crypto.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x64-windows/debug/lib/ssl.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x64-windows/debug/lib/tls.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x64-windows/lib/crypto.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x64-windows/lib/ssl.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x64-windows/lib/tls.lib': operation not permitted
-- Performing post-build validation
The following EXEs were found in /bin or /debug/bin. EXEs are not valid distribution targets.

    F:/11949/vcpkg/packages/libressl_x64-windows/bin/ocspcheck.exe
    F:/11949/vcpkg/packages/libressl_x64-windows/bin/openssl.exe

Found 1 error(s). Please correct the portfile:

Could you please try to fix this?

@Maximus5
Copy link
Copy Markdown
Contributor Author

It works with tools now

@NancyLi1013
Copy link
Copy Markdown
Contributor

It can be built with tools successfully.
But there are still some CMake errors like this:

PS F:\11949\vcpkg> .\vcpkg.exe install libressl[tools]:x64-windows
Computing installation plan...
The following packages will be built and installed:
libressl[core,tools]:x64-windows
Starting package 1/1: libressl:x64-windows
Building package libressl[core,tools]:x64-windows...
-- Using cached F:/11949/vcpkg/downloads/libressl-2.9.1.tar.gz
-- Using source at F:/11949/vcpkg/buildtrees/libressl/src/2.9.1-793e310523
-- Configuring x64-windows
-- Building x64-windows-dbg
-- Building x64-windows-rel
-- Installing: F:/11949/vcpkg/packages/libressl_x64-windows/share/libressl/copyright
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x64-windows/debug/lib/crypto.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x64-windows/debug/lib/ssl.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x64-windows/debug/lib/tls.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x64-windows/lib/crypto.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x64-windows/lib/ssl.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x64-windows/lib/tls.lib': operation not permitted
-- Performing post-build validation
-- Performing post-build validation done
Building package libressl[core,tools]:x64-windows... done
Installing package libressl[core,tools]:x64-windows...
Installing package libressl[core,tools]:x64-windows... done
Elapsed time for package libressl:x64-windows: 7.986 min

Total elapsed time: 7.991 min

@Maximus5
Copy link
Copy Markdown
Contributor Author

That operation not permitted stuff happens with only core even on master branch.
Should I fix that in this PR?

@Maximus5
Copy link
Copy Markdown
Contributor Author

Maximus5 commented Jul 13, 2020

@NancyLi1013 Well, I've made investigations and this stuff

CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x64-windows/debug/lib/crypto.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x64-windows/debug/lib/ssl.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x64-windows/debug/lib/tls.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x64-windows/lib/crypto.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x64-windows/lib/ssl.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x64-windows/lib/tls.lib': operation not permitted

is because of the bug in vcpkg/stl.

F:/11949/vcpkg/packages/libressl_x64-windows/lib/crypto.lib is actually a symlink to the F:/11949/vcpkg/packages/libressl_x64-windows/lib/crypto-45.lib (versioned lib in the same folder).
And here happens:

  • install_files_and_write_listfile calls RealFileSystem::copy_symlink which in turns calls fs::stdfs::copy_symlink
  • std::experimental::filesystem::copy_symlink calls std::experimental::filesystem::create_symlink where
_Oldpval=F:\11949\vcpkg\packages\libressl_x64-windows\lib\crypto.lib
_Newpval=F:\11949\vcpkg\installed\x64-windows\debug\lib\crypto.lib
  • which calls __crtCreateSymbolicLinkW and in finally CreateSymbolicLinkW but without SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE flag.

From my point of view, there is two problems here and both are out of scope of this PR

  1. vcpkg calls std::experimental::filesystem::create_symlink which is expected to be runned in elevated processes only (no SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE flag)
  2. vcpkg tries to create a symbolic link targeted into packages folder. I'm not sure if this is a bug or not. But we already copied (or are going to copy) the crypto-45.lib into installed folder and probably the symbolic link should point to this new file in the installed folder.

@NancyLi1013
Copy link
Copy Markdown
Contributor

Hi @Maximus5
Could you please merge master to this PR?

@NancyLi1013
Copy link
Copy Markdown
Contributor

/azp run

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

@Maximus5
Copy link
Copy Markdown
Contributor Author

Hi
Could this one be merged?

Comment thread ports/libressl/CONTROL Outdated
@NancyLi1013
Copy link
Copy Markdown
Contributor

NancyLi1013 commented Jul 24, 2020

Hi @Maximus5
The error still exists. Do we need to fix it?

CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x86-windows/debug/lib/crypto.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x86-windows/debug/lib/ssl.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x86-windows/debug/lib/tls.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x86-windows/lib/crypto.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x86-windows/lib/ssl.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x86-windows/lib/tls.lib': operation not permitted

On master, it seems there is no this error.

PS F:\tools\vcpkg> .\vcpkg install libressl[tools]
Computing installation plan...
The following packages will be built and installed:
    libressl[core,tools]:x86-windows
Detecting compiler hash for triplet x86-windows...
Starting package 1/1: libressl:x86-windows
Building package libressl[core,tools]:x86-windows...
Building package libressl[core,tools]:x86-windows... done
Installing package libressl[core,tools]:x86-windows...
Installing package libressl[core,tools]:x86-windows... done
Elapsed time for package libressl:x86-windows: 210.4 ms

Total elapsed time: 3.885 s

@Maximus5
Copy link
Copy Markdown
Contributor Author

Strange. I had no such errors locally (Windows 10).
I've merged origin/master again, but it looks like proper commits were already there.

> vcpkg install --recurse libressl[core,tools]:x64-windows
Computing installation plan...
The following packages will be built and installed:
    libressl[core,tools]:x64-windows
Detecting compiler hash for triplet x64-windows...
Starting package 1/1: libressl:x64-windows
Building package libressl[core,tools]:x64-windows...
Could not locate cached archive: C:\Users\...\AppData\Local\vcpkg\archives\ad\ada4f0ea7a07a670352572f97379e9394a2603be.zip
-- Using cached C:/SRC/vcpkg/downloads/libressl-2.9.1.tar.gz
-- Cleaning sources at C:/SRC/vcpkg/buildtrees/libressl/src/2.9.1-793e310523.clean. Use --editable to skip cleaning for the packages you specify.
-- Extracting source C:/SRC/vcpkg/downloads/libressl-2.9.1.tar.gz
-- Applying patch 0001-enable-ocspcheck-on-msvc.patch
-- Applying patch 0002-suppress-msvc-warnings.patch
-- Using source at C:/SRC/vcpkg/buildtrees/libressl/src/2.9.1-793e310523.clean
-- Configuring x64-windows
-- Building x64-windows-dbg
-- Building x64-windows-rel
-- Installing: C:/SRC/vcpkg/packages/libressl_x64-windows/share/libressl/copyright
-- Performing post-build validation
-- Performing post-build validation done
Stored binary cache: C:\Users\...\AppData\Local\vcpkg\archives\ad\ada4f0ea7a07a670352572f97379e9394a2603be.zip
Building package libressl[core,tools]:x64-windows... done
Installing package libressl[core,tools]:x64-windows...
Installing package libressl[core,tools]:x64-windows... done
Elapsed time for package libressl:x64-windows: 2.335 min

Total elapsed time: 2.381 min

> vcpkg install libressl[tools]
Computing installation plan...
The following packages will be built and installed:
    libressl[core,tools]:x86-windows
Detecting compiler hash for triplet x86-windows...
Starting package 1/1: libressl:x86-windows
Building package libressl[core,tools]:x86-windows...
Could not locate cached archive: C:\Users\...\AppData\Local\vcpkg\archives\d0\d0e5764689e0f73dd9e5ed0878dd33728100f0c1.zip
-- Using cached C:/SRC/vcpkg/downloads/libressl-2.9.1.tar.gz
-- Cleaning sources at C:/SRC/vcpkg/buildtrees/libressl/src/2.9.1-793e310523.clean. Use --editable to skip cleaning for the packages you specify.
-- Extracting source C:/SRC/vcpkg/downloads/libressl-2.9.1.tar.gz
-- Applying patch 0001-enable-ocspcheck-on-msvc.patch
-- Applying patch 0002-suppress-msvc-warnings.patch
-- Using source at C:/SRC/vcpkg/buildtrees/libressl/src/2.9.1-793e310523.clean
-- Configuring x86-windows
-- Building x86-windows-dbg
-- Building x86-windows-rel
-- Installing: C:/SRC/vcpkg/packages/libressl_x86-windows/share/libressl/copyright
-- Performing post-build validation
-- Performing post-build validation done
Stored binary cache: C:\Users\...\AppData\Local\vcpkg\archives\d0\d0e5764689e0f73dd9e5ed0878dd33728100f0c1.zip
Building package libressl[core,tools]:x86-windows... done
Installing package libressl[core,tools]:x86-windows...
Installing package libressl[core,tools]:x86-windows... done
Elapsed time for package libressl:x86-windows: 2.327 min

Total elapsed time: 2.378 min

@azure-pipelines
Copy link
Copy Markdown

Commenter does not have sufficient privileges for PR 11949 in repo microsoft/vcpkg

@Maximus5
Copy link
Copy Markdown
Contributor Author

Could you recheck this please?

@NancyLi1013
Copy link
Copy Markdown
Contributor

NancyLi1013 commented Jul 28, 2020

It might be something with my local environment. I will check this again.

Let's waiting for CI checks. Once all passed, it can be merged.

Edit: I test this on my another machine, but there is still the same error.
Sorry, I don't know why this happened.

@Maximus5
Copy link
Copy Markdown
Contributor Author

Maximus5 commented Jul 28, 2020

started 3d 10h 35m 15s ago

Doesn't that mean something wrong is with CI?

@NancyLi1013
Copy link
Copy Markdown
Contributor

Yes, there is something wrong with our CI.
I will try to re run this PR after it is fine.

@NancyLi1013
Copy link
Copy Markdown
Contributor

/azp run

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

@NancyLi1013
Copy link
Copy Markdown
Contributor

@strega-nil
Could you please help take a look if the error info will block this PR merged?

CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x64-windows/debug/lib/crypto.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x64-windows/debug/lib/ssl.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x64-windows/debug/lib/tls.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x64-windows/lib/crypto.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x64-windows/lib/ssl.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x64-windows/lib/tls.lib': operation not permitted

@NancyLi1013 NancyLi1013 added info:needs-maintainer-attention Lets the current 'on rotation' vcpkg maintainer know they need to look at this. requires:discussion labels Jul 31, 2020
@Maximus5
Copy link
Copy Markdown
Contributor Author

Maximus5 commented Jul 31, 2020

@NancyLi1013 Are you sure you are using fresh vcpkg.exe? For me similar error appears only if I build vcpkg from 830893f and before (without this PR changes definitely).
But in either case, the error differs. My output from old vcpkg is

vcpkg install --recurse libressl[core,tools]:x86-windows
Computing installation plan...
The following packages will be built and installed:
    libressl[core,tools]:x86-windows
Detecting compiler hash for triplet x86-windows...
Starting package 1/1: libressl:x86-windows
Building package libressl[core,tools]:x86-windows...
Could not locate cached archive: C:\Users\...\AppData\Local\vcpkg\archives\c4\c485cb423b25342d41a422d98d82f5af825513dd.zip
-- Using cached C:/SRC/vcpkg/downloads/libressl-2.9.1.tar.gz
-- Cleaning sources at C:/SRC/vcpkg/buildtrees/libressl/src/2.9.1-793e310523.clean. Use --editable to skip cleaning for the packages you specify.
-- Extracting source C:/SRC/vcpkg/downloads/libressl-2.9.1.tar.gz
-- Applying patch 0001-enable-ocspcheck-on-msvc.patch
-- Applying patch 0002-suppress-msvc-warnings.patch
-- Using source at C:/SRC/vcpkg/buildtrees/libressl/src/2.9.1-793e310523.clean
-- Configuring x86-windows
-- Building x86-windows-dbg
-- Building x86-windows-rel
-- Installing: C:/SRC/vcpkg/packages/libressl_x86-windows/share/libressl/copyright
-- Performing post-build validation
-- Performing post-build validation done
Stored binary cache: C:\Users\...\AppData\Local\vcpkg\archives\c4\c485cb423b25342d41a422d98d82f5af825513dd.zip
Building package libressl[core,tools]:x86-windows... done
Installing package libressl[core,tools]:x86-windows...
failed: C:\SRC\vcpkg\installed\x86-windows\debug\lib\crypto.lib: A required privilege is not held by the client.
failed: C:\SRC\vcpkg\installed\x86-windows\debug\lib\ssl.lib: A required privilege is not held by the client.
failed: C:\SRC\vcpkg\installed\x86-windows\debug\lib\tls.lib: A required privilege is not held by the client.
failed: C:\SRC\vcpkg\installed\x86-windows\lib\crypto.lib: A required privilege is not held by the client.
failed: C:\SRC\vcpkg\installed\x86-windows\lib\ssl.lib: A required privilege is not held by the client.
failed: C:\SRC\vcpkg\installed\x86-windows\lib\tls.lib: A required privilege is not held by the client.
Installing package libressl[core,tools]:x86-windows... done
Elapsed time for package libressl:x86-windows: 2.542 min

What OS and compiler is that?

CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x64-windows/debug/lib/crypto.lib': operation not permitted

You have CMake Error: failed to create symbolic link message, mine is only failed: . And back/forward slashes differs too.

Also I have only VS 2019 installed on my PC.

@NancyLi1013
Copy link
Copy Markdown
Contributor

Hi @Maximus5

I clone a new repro to build this and there is still the same error. I can confirm I have got the latest commit.

PS F:\11949\vcpkg> .\vcpkg.exe install libressl[tools]
Computing installation plan...
The following packages will be built and installed:
    libressl[core,tools]:x86-windows
Detecting compiler hash for triplet x86-windows...
Starting package 1/1: libressl:x86-windows
Building package libressl[core,tools]:x86-windows...
Could not locate cached archive: C:\Users\v-ruil\AppData\Local\vcpkg\archives\28\283a0f178f858a54446ccb29a0160bbf57973d6b.zip
-- Using cached F:/11949/vcpkg/downloads/libressl-2.9.1.tar.gz
-- Cleaning sources at F:/11949/vcpkg/buildtrees/libressl/src/2.9.1-793e310523.clean. Use --editable to skip cleaning for the packages you specify.
-- Extracting source F:/11949/vcpkg/downloads/libressl-2.9.1.tar.gz
-- Applying patch 0001-enable-ocspcheck-on-msvc.patch
-- Applying patch 0002-suppress-msvc-warnings.patch
-- Using source at F:/11949/vcpkg/buildtrees/libressl/src/2.9.1-793e310523.clean
-- Configuring x86-windows
-- Building x86-windows-dbg
-- Building x86-windows-rel
-- Installing: F:/11949/vcpkg/packages/libressl_x86-windows/share/libressl/copyright
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x86-windows/debug/lib/crypto.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x86-windows/debug/lib/ssl.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x86-windows/debug/lib/tls.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x86-windows/lib/crypto.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x86-windows/lib/ssl.lib': operation not permitted
CMake Error: failed to create symbolic link 'F:/11949/vcpkg/packages/libressl_x86-windows/lib/tls.lib': operation not permitted
-- Performing post-build validation
-- Performing post-build validation done
Stored binary cache: C:\Users\v-ruil\AppData\Local\vcpkg\archives\28\283a0f178f858a54446ccb29a0160bbf57973d6b.zip
Building package libressl[core,tools]:x86-windows... done
Installing package libressl[core,tools]:x86-windows...
Installing package libressl[core,tools]:x86-windows... done
Elapsed time for package libressl:x86-windows: 2.282 min

Total elapsed time: 2.341 min

I try to build this on two machines and they failed with the same error. One is Visual Studio 2017, another is Vistudio 2019. Both are on Windows 10.

I don't know what caused this error info.

@strega-nil
Copy link
Copy Markdown
Contributor

strega-nil commented Aug 3, 2020

This is really unfortunate; it looks like libressl requires permissions to create symbolic links (cc @BillyONeal for an example of a port that does this). I'm not sure how to solve this; it means that we can't currently deal with libressl on Windows.

However, it is an existing error. I'd recommend marking it Supports: !windows until we fix our symlink support.

Also, fwiw, they don't even support building with VS, so Supports: !windows is probably the best solution for now. If someone wants to, they can allow support for mingw.

@BillyONeal
Copy link
Copy Markdown
Member

@strega-nil This isn't 'our symlink support'; it's blowing up inside the port itself's build script. It will probably work if it gets run as admin.

@strega-nil
Copy link
Copy Markdown
Contributor

@BillyONeal this is true, but we don't support symlinks in bootstrapped vcpkg anyways.

@strega-nil strega-nil merged commit c74bc92 into microsoft:master Aug 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category:port-feature The issue is with a library, which is requesting new capabilities that didn’t exist info:needs-maintainer-attention Lets the current 'on rotation' vcpkg maintainer know they need to look at this.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants